Techniques For Generation Of A Conformant Output Sub-Bitstream

ABSTRACT

Examples of video encoding methods and apparatus and video decoding methods and apparatus are described. An example method of video processing includes performing a conversion between a video including multiple layers and a bitstream of the video according to a rule, wherein the rule specifies that, in a first process of sub-bitstream extraction to output a first output sub-bitstream, the first output sub-bitstream is extracted without removing network abstraction layer (NAL) units of a particular type and having a particular NAL unit header identifier value, and wherein the particular type includes an access unit delimiter (AUD) NAL unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/992,176, filed on Nov. 22, 2022, which is a continuation ofInternational Patent Application No. PCT/US2021/033648, filed on May 21,2021, which claims the priority to and benefits of U.S. ProvisionalPatent Application No. 63/029,308, filed on May 22, 2020. All theaforementioned patent applications are hereby incorporated by referencein their entireties.

TECHNICAL FIELD

The present disclosure relates to image and video coding and decoding.

BACKGROUND

Digital video accounts for the largest bandwidth use on the internet andother digital communication networks. As the number of connected userdevices capable of receiving and displaying video increases, it isexpected that the bandwidth demand for digital video usage will continueto grow.

SUMMARY

The present disclosure discloses embodiments that can be used by videoencoders and decoders to perform video encoding or decoding.

In one example aspect, a video processing method is disclosed. Themethod includes performing a conversion between a video comprisingmultiple layers and a bitstream of the video according to a rule,wherein the rule specifies a maximum allowed value of a temporal layeridentifier value of a sub-bitstream that is extracted from thebitstream.

In another example aspect, a video processing method is disclosed. Themethod includes performing a conversion between a video and a bitstreamof the video according to a rule, wherein the rule defines networkabstraction layer (NAL) units to be extracted from the bitstream duringa sub-bitstream extraction process to output a sub-bitstream, andwherein the rule specifies to derive the sub-bitstream based on whethera list of NAL unit header identifier values in an output layer set (OLS)with a target OLS index does not include all values of NAL unit headeridentifiers in all video coding layer (VCL) NAL units in the bitstreamthat is input to the sub -bitstream extraction process.

In another example aspect, a video processing method is disclosed. Themethod includes performing a conversion between a video and a bitstreamof the video according to a rule, wherein the rule defines networkabstraction layer (NAL) units to be extracted from the bitstream duringa sub-bitstream extraction process to output a sub-bitstream, andwherein the rule specifies, responsive to a payload type of a firstsupplemental enhancement information (SEI) message included in an SEInetwork abstraction layer (NAL) unit, to disallow the SEI NAL unit tocontain an SEI message with a certain payload type.

In another example aspect, a video processing method is disclosed. Themethod includes performing a conversion between a video and a bitstreamof the video according to a format rule, wherein the format rulespecifies that the bitstream includes a flag specifying whether one ormore non-scalable nested supplemental enhancement information (SEI)messages with one or more particular payload types apply to all outputlayer sets referenced by a coding layer.

In another example aspect, a video processing method is disclosed. Themethod includes performing a conversion between a video and a bitstreamof the video according to a rule, wherein the rule specifies asub-bitstream extraction process by which an output sub-bitstream isextracted from the bitstream without removing network abstraction layer(NAL) units of a particular type and having a particular NAL unit headeridentifier value, wherein the particular type includes an access unitdelimiter (AUD) NAL unit.

In another example aspect, a video processing method is disclosed. Themethod includes performing a conversion between a video including one ormore layers and a bitstream of the video according to a rule, whereinthe rule specifies, in a process of sub-bitstream extraction, to removenetwork abstraction layer (NAL) units that include a scalable nestingsupplementary enhancement information (SEI) message applied to layersthat are not included in a target output layer set (OLS).

In another example aspect, a video processing method is disclosed. Themethod includes performing a conversion between a video including one ormore layers and a bitstream of the video according to a rule, whereinthe rule specifies, in a process of sub-bitstream extraction, anon-scalable nested supplementary enhancement information (SEI) messageis generated by extracting an SEI message that is scalably-nested from ascalable nesting SEI message based on a first flag indicating whetherthe SEI message applies to specific output layer sets (OLSs) and asecond flag indicating whether the SEI message applies to allsubpictures or only to specific subpictures.

In another example aspect, a video processing method is disclosed. Themethod includes performing a conversion between a video including one ormore layers and a bitstream of the video according to a rule, whereinthe rule specifies, in a process of sub-bitstream extraction, anon-scalable nested supplemental enhancement information (SEI) messageis generated by extracting multiple scalable-nested SEI messages from afirst SEI network abstraction layer (NAL) unit in a picture unit.

In another example aspect, a video processing method is disclosed. Themethod includes performing a conversion between a video including one ormore layers and a bitstream of the video according to a rule, whereinthe rule specifies a sub-bitstream extraction process to generate anoutput bitstream, wherein the rule specifies handling of one or moresupplemental enhancement information (SEI) network abstraction layer(NAL) units during the sub-bitstream extraction process.

In yet another example aspect, a video encoder apparatus is disclosed.The video encoder comprises a processor configured to implementabove-described methods.

In yet another example aspect, a video decoder apparatus is disclosed.The video decoder comprises a processor configured to implementabove-described methods.

In yet another example aspect, a computer readable medium having codestored thereon is disclosed. The code embodies one of the methodsdescribed herein in the form of processor-executable code.

These, and other, features are described throughout the presentdisclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram that illustrates a video coding system inaccordance with some embodiments of the present disclosure.

FIG. 2 is a block diagram of an example hardware platform used for videoprocessing.

FIG. 3 is a flowchart for an example method of video processing.

FIG. 4 is a block diagram that illustrates an example video codingsystem.

FIG. 5 is a block diagram that illustrates an encoder in accordance withsome embodiments of the present disclosure.

FIG. 6 is a block diagram that illustrates a decoder in accordance withsome embodiments of the present disclosure.

FIGS. 7A to 7D are flowcharts for example methods of video processingbased on some embodiments of the present disclosure.

FIG. 8 is a flowchart for an example method of video processing based onsome embodiments of the present disclosure.

FIG. 9 is a flowchart for an example method of video processing based onsome embodiments of the present disclosure.

FIGS. 10A to 10C are flowcharts for example methods of video processingbased on some embodiments of the present disclosure.

DETAILED DESCRIPTION

Section headings are used in the present disclosure for ease ofunderstanding and do not limit the applicability of embodimentsdisclosed in each section only to that section. Furthermore, H.266terminology is used in some description only for ease of understandingand not for limiting scope of the disclosed embodiments. As such, theembodiments described herein are applicable to other video codecprotocols and designs also.

1. Introduction

This disclosure is related to video coding technologies. Specifically,it is about some improvements on the general sub-bitstream extractionprocess, signalling of picture-level hypothetical reference decoder(HRD) parameters, and containing supplemental enhancement information(SEI) messages in SEI network abstraction layer (NAL) units. The ideasmay be applied individually or in various combination, to any videocoding standard or non-standard video codec that supports multi-layervideo coding, e.g., the being-developed Versatile Video Coding (VVC).

2. Abbreviations

APS Adaptation Parameter Set

AU Access Unit

AUD Access Unit Delimiter

AVC Advanced Video Coding

CLVS Coded Layer Video Sequence

CPB Coded Picture Buffer

CRA Clean Random Access

CTU Coding Tree Unit

CVS Coded Video Sequence

DCI Decoding Capability Information

DPB Decoded Picture Buffer

EOB End Of Bitstream

EOS End Of Sequence

GDR Gradual Decoding Refresh

HEVC High Efficiency Video Coding

HRD Hypothetical Reference Decoder

IDR Instantaneous Decoding Refresh

ILP Inter-Layer Prediction

ILRP Inter-Layer Reference Picture

IRAP Intra Random Access Points

JEM Joint Exploration Model

LTRP Long-Term Reference Picture

MCTS Motion-Constrained Tile Sets

NAL Network Abstraction Layer

OLS Output Layer Set

PH Picture Header

PPS Picture Parameter Set

PTL Profile, Tier and Level

PU Picture Unit

RAP Random Access Point

RBSP Raw Byte Sequence Payload

SEI Supplemental Enhancement Information

SPS Sequence Parameter Set

STRP Short-Term Reference Picture

SVC Scalable Video Coding

VCL Video Coding Layer

VPS Video Parameter Set

VTM VVC Test Model

VUI Video Usability Information

VVC Versatile Video Coding

3. Initial Discussion

Video coding standards have evolved primarily through the development ofthe well-known International Telecommunication Union (ITU)Telecommunication Standardization Sector (ITU-T) and InternationalOrganization for Standardization (ISO)/ International ElectrotechnicalCommission (IEC) standards. The ITU-T produced H.261 and H.263, ISO/IECproduced Moving Picture Experts Group (MPEG)-1 and MPEG-4 Visual, andthe two organizations jointly produced the H.262/MPEG-2 Video andH.264/MPEG-4 Advanced Video Coding (AVC) and H.265/HEVC standards. SinceH.262, the video coding standards are based on the hybrid video codingstructure wherein temporal prediction plus transform coding areutilized. To explore the future video coding technologies beyond HEVC,the Joint Video Exploration Team (JVET) was founded by Video CodingExperts Group (VCEG) and MPEG jointly in 2015. Since then, many newmethods have been adopted by JVET and put into the reference softwarenamed Joint Exploration Model (JEM). The JVET meeting is concurrentlyheld once every quarter, and the new coding standard is targeting a 50%bitrate reduction as compared to HEVC. The new video coding standard wasofficially named as Versatile Video Coding (VVC) in the April 2018 JVETmeeting, and the first version of VVC test model (VTM) was released atthat time. As there are continuous effort contributing to VVCstandardization, new coding techniques are being adopted to the VVCstandard in every JVET meeting. The VVC working draft and test model VTMare then updated after every meeting. The VVC project is now aiming fortechnical completion (FDIS) at the July 2020 meeting.

3.1. Picture Resolution Change Within a Sequence

In AVC and HEVC, the spatial resolution of pictures cannot change unlessa new sequence using a new SPS starts, with an intra random accesspoints (IRAP) picture. VVC enables picture resolution change within asequence at a position without encoding an IRAP picture, which is alwaysintra-coded. This feature is sometimes referred to as reference pictureresampling (RPR), as the feature needs resampling of a reference pictureused for inter prediction when that reference picture has a differentresolution than the current picture being decoded.

The scaling ratio is restricted to be greater than or equal to ½ (2times downsampling from the reference picture to the current picture),and less than or equal to 8 (8 times upsampling). Three sets ofresampling filters with different frequency cutoffs are specified tohandle various scaling ratios between a reference picture and thecurrent picture. The three sets of resampling filters are appliedrespectively for the scaling ratio ranging from ½ to 1/1.75, from 1/1.75to 1/1.25, and from 1/1.25 to 8. Each set of resampling filters has 16phases for luma and 32 phases for chroma which is same to the case ofmotion compensation interpolation filters. Actually, the normal MCinterpolation process is a special case of the resampling process withscaling ratio ranging from 1/1.25 to 8. The horizontal and verticalscaling ratios are derived based on picture width and height, and theleft, right, top and bottom scaling offsets specified for the referencepicture and the current picture.

Other aspects of the VVC design for support of this feature that aredifferent from HEVC include: i) the picture resolution and thecorresponding conformance window are signalled in the PPS instead of inthe SPS, while in the SPS the maximum picture resolution is signalled;and ii) for a single-layer bitstream, each picture store (a slot in theDPB for storage of one decoded picture) occupies the buffer size asrequired for storing a decoded picture having the maximum pictureresolution.

3.2. Scalable Video Coding (SVC) in General and in VVC

Scalable video coding (SVC, sometimes also just referred to asscalability in video coding) refers to video coding in which a baselayer (BL), sometimes referred to as a reference layer (RL), and one ormore scalable enhancement layers (ELs) are used. In SVC, the base layercan carry video data with a base level of quality. The one or moreenhancement layers can carry additional video data to support, forexample, higher spatial, temporal, and/or signal-to-noise (SNR) levels.Enhancement layers may be defined relative to a previously encodedlayer. For example, a bottom layer may serve as a BL, while a top layermay serve as an EL. Middle layers may serve as either ELs or RLs, orboth. For example, a middle layer (e.g., a layer that is neither thelowest layer nor the highest layer) may be an EL for the layers belowthe middle layer, such as the base layer or any intervening enhancementlayers, and at the same time serve as a RL for one or more enhancementlayers above the middle layer. Similarly, in the multiview orthree-dimensional (3D) extension of the HEVC standard, there may bemultiple views, and information of one view may be utilized to code(e.g., encode or decode) the information of another view (e.g., motionestimation, motion vector prediction and/or other redundancies).

In SVC, the parameters used by the encoder or the decoder are groupedinto parameter sets based on the coding level (e.g., video-level,sequence-level, picture-level, slice level, etc.) in which they may beutilized. For example, parameters that may be utilized by one or morecoded video sequences of different layers in the bitstream may beincluded in a video parameter set (VPS), and parameters that areutilized by one or more pictures in a coded video sequence may beincluded in a sequence parameter set (SPS). Similarly, parameters thatare utilized by one or more slices in a picture may be included in apicture parameter set (PPS), and other parameters that are specific to asingle slice may be included in a slice header. Similarly, theindication of which parameter set(s) a particular layer is using at agiven time may be provided at various coding levels.

Thanks to the support of reference picture resampling (RPR) in VVC,support of a bitstream containing multiple layers, e.g., two layers withstandard definition (SD) and high definition (HD) resolutions in VVC canbe designed without the need any additional signal-processing-levelcoding tool, as upsampling needed for spatial scalability support canjust use the RPR upsampling filter. Nevertheless, high-level syntaxchanges (compared to not supporting scalability) are needed forscalability support. Scalability support is specified in VVC version 1.Different from the scalability supports in any earlier video codingstandards, including in extensions of AVC and HEVC, the design of VVCscalability has been made friendly to single-layer decoder designs asmuch as possible. The decoding capability for multi-layer bitstreams arespecified in a manner as if there were only a single layer in thebitstream. For example, the decoding capability, such as DPB size, isspecified in a manner that is independent of the number of layers in thebitstream to be decoded. Basically, a decoder designed for single-layerbitstreams does not need much change to be able to decode multi-layerbitstreams. Compared to the designs of multi-layer extensions of AVC andHEVC, the hypertext transfer protocol live streaming (HLS) aspects havebeen significantly simplified at the sacrifice of some flexibilities.For example, an IRAP AU is required to contain a picture for each of thelayers present in the CVS.

3.3. Parameter Sets

AVC, HEVC, and VVC specify parameter sets. The types of parameter setsinclude SPS, PPS, APS, and VPS. SPS and PPS are supported in all of AVC,HEVC, and VVC. VPS was introduced since HEVC and is included in bothHEVC and VVC. APS was not included in AVC or HEVC but is included in thelatest VVC draft text.

SPS was designed to carry sequence-level header information, and PPS wasdesigned to carry infrequently changing picture-level headerinformation. With SPS and PPS, infrequently changing information neednot to be repeated for each sequence or picture, hence redundantsignalling of this information can be avoided. Furthermore, the use ofSPS and PPS enables out-of-band transmission of the important headerinformation, thus not only avoiding the need for redundant transmissionsbut also improving error resilience.

VPS was introduced for carrying sequence-level header information thatis common for all layers in multi-layer bitstreams.

APS was introduced for carrying such picture-level or slice-levelinformation that needs quite some bits to code, can be shared bymultiple pictures, and in a sequence there can be quite many differentvariations.

3.4. General Sub-Bitstream Extraction Process

Clause C.6 of the latest VVC text specifies a general sub-bitstreamextraction process, as follows:

C.6 Sub-Bitstream Extraction Process

Inputs to this process are a bitstream inBitstream, a target OLS indextargetOlsIdx, and a target highest Temporand value tIdTarget.

Output of this process is a sub-bitstream outBitstream.

It is a requirement of bitstream conformance for the input bitstreamthat any output sub-bitstream that satisfies all of the followingconditions shall be a conforming bitstream:

-   -   The output sub-bitstream is the output of the process specified        in this clause with the bitstream, targetOlsIdx equal to an        index to the list of OLSs specified by the VPS, and tIdTarget        equal to any value in the range of 0 to 6, inclusive, as inputs.    -   The output sub-bitstream contains at least one VCL NAL unit with        nuh_layer_id equal to each of the nuh_layer_id values in        LayerIdInOls[targetOlsIdx].    -   The output sub-bitstream contains at least one VCL NAL unit with        Temporand equal to tIdTarget.    -   NOTE—A conforming bitstream contains one or more coded slice NAL        units with Temporand equal to 0, but does not have to contain        coded slice NAL units with nuh_layer_id equal to 0.

The output sub-bitstream OutBitstream is derived as follows:

-   -   1. The bitstream outBitstream is set to be identical to the        bitstream inBitstream.    -   2. Remove from outBitstream all NAL units with Temporand greater        than tIdTarget.    -   3. Remove from outBitstream all NAL units with nal_unit_type not        equal to any of VPS_NUT, DCI_NUT, and EOB_NUT and with        nuh_layer_id not included in the list LayerIdInOls targetOlsIdx        1.    -   4. Remove from outBitstream all VCL NAL units for which all of        the following conditions are true, and their associated non-VCL        NAL units with nal_unit_type equal to PH_NUT, FD_NUT,        SUFFIX_SEI_NUT, and PREFIX_SEI_NUT with PayloadType not equal to        0, 1, or 130:        -   nal_unit_type is equal to TRAIL_NUT, STSA_NUT, RADL_NUT, or            RASL_NUT, or nal_unit_type is equal to GDR_NUT and the            associated ph_recovery_poc_cnt is not equal to 0.        -   nuh_layer_id is equal to LayerIdInOls[targetOlsIdx][j] for a            value of j in the range of 0 to            NumLayersInOls[targetOlsIdx]−1 inclusive.        -   Temporand is greater than or equal to            NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]].    -   5. Remove from outBitstream all SEI NAL units that contain a        scalable nesting SEI message that has sn_ols_flag equal to 1 and        there is no value of i in the range of 0 to sn_num_olss_minus1,        inclusive, such that NestingOlsIdx[i] is equal to targetOlsIdx.    -   6. When LayerIdInOls[targetOlsIdx] does not include all values        of nuh_layer_id in all NAL units in the bitstream, the following        applies:        -   a. Remove from outBitstream all SEI NAL units that contain a            non-scalable-nested SEI message with payloadType equal to 0            (BP) or 130 (DUI).        -   b. When general_same_pic_timing_in_all_ols_flag is equal to            0, remove from outBitstream all SEI NAL units that contain a            non-scalable-nested SEI message with payloadType equal to 1            (PT).        -   c. When outBitstream contains SEI NAL units that contain a            scalable nesting SEI message with sn_ols_flag equal to 1 and            are applicable to outBitstream (NestingOlsIdx[i] is equal to            targetOlsIdx), the following applies:            -   If general_same_pic_timing_in_all_ols_flag is equal to                0, extract appropriate non-scalable-nested SEI message                with payloadType equal to 0 (BP), 1 (PT), or 130 (DUI)                from the scalable nesting SEI message and include those                SEI messages in outBitstream.            -   Otherwise (general_same_pic_timing_in_all_ols_flag is                equal to 1), extract appropriate non-scalable-nested SEI                message with payloadType equal to 0 (BP) or 130 (DUI)                from the scalable nesting SEI message and include those                SEI messages in outBitstream.

4. Technical Problems Solved By Disclosed Technical Solutions

The existing designs of the general sub-bitstream extraction process andrelated other parts in the latest VVC text (in JVET-R2001-vA/v10) havethe following problems:

-   -   1) In the conditions under which an output sub-bitstream is        required to be a conforming bitstream, the value of tIdTarget is        said to be in the range of 0 to 6, inclusive. However, in many        bitstreams, the highest Temporand value is less than 6, and that        value is specified by the syntax element vps_max_sublayers_minus        1.    -   2) An access unit delimiter (AUD) NAL unit, when present, can        have any nuh_layer_id value.

However, step 3 of the sub-bitstream extraction process would remove theAUD NAL units for which the nuh_layer_id values are not included in thelist LayerIdInOls[targetOlsIdx].

-   -   3) Some SEI NAL units contain a scalable nesting SEI message        with sn_ols_flag equal to 0 while the applicable layers as        indicted in the scalable nesting SEI message do not include any        layer in the target OLS, i.e., none of the applicable layers'        nuh_layer_id values is not included in the list        LayerIdInOls[targetOlsIdx]. These SEI NAL units should also be        removed.    -   4) The condition of step 6, i.e., “When        LayerIdInOls[targetOlsIdx] does not include all values of        nuh_layer_id in all NAL units in the bitstream” has the        following two issues.        -   a. The condition does not work for cases when DCI, VPS, AUD,            or EOB NAL units are present and have nuh_layer_id not equal            to any of the nuh_layer_id values of the VCL NAL units.        -   b. The phrase “the bitstream” is not clear, as there are two            bitstreams involved in the context, inBitstream and            outBitstream.    -   5) Step 6.c would extract scalable-nested SEI messages, to        generate non-scalable-nested SEI messages, from scalable nesting        SEI messages with both sn_ols_flag equal to 1 and sn_subpic_flag        equal to 1, while such scalable-nested SEI messages only apply        to specific subpictures and thus should not be extracted.    -   6) In step 6.c, when multiple scalable-nested SEI messages are        extracted from one SEI NAL unit seiNalUnitA to be        non-scalable-nested SEI messages, they should still be included        in one SEI NAL unit seiNalUnitB, and the SEI NAL unit        seiNalUnitB should be included in the same PU that contained the        SEI NAL unit seiNalUnitA. However, this is not specified.    -   7) Step 6.c should remove, from outBitstream, all SEI NAL units        from which some SEI messages have been extracted and included as        non-scalable-nested SEI messages. However, this is not        specified.    -   8) A constraint is lacking such that when an SEI NAL unit        contains an SEI message with payloadType equal to 0, 1, or 130,        the SEI NAL unit shall not contain an SEI message with        payloadType not equal to 0 (buffering period (BP)), 1 (picture        timing (PT)), 130 (decoding unit information (DUI)), or 133        (scalable nesting). This causes the removal of SEI messages in        step 4 involves more than just removal of SEI NAL units.    -   9) The flag general_same_pic_timing_in_all_ols_flag only        specifies whether non-scalable-nested PT SEI messages apply to        all OLSs. However, information carried in the DUI SEI messages        are for similar purposes as in the PT SEI messages.        5. A listing of Technical Solutions and Embodiments

To solve the above problems, and others, methods as summarized below aredisclosed. The items should be considered as examples to explain thegeneral concepts and should not be interpreted in a narrow way.Furthermore, these items can be applied individually or combined in anymanner.

-   -   1) To solve problem 1, the conditions under which an output        sub-bitstream is required to be a conforming bitstream are        specified such that the value of tIdTarget is specified be in        the range of 0 to vps_max_sublayers_minus1, inclusive.        -   a. Alternatively, the conditions under which an output            sub-bitstream is required to be a conforming bitstream are            specified such that the value of tIdTarget is specified be            in the range of 0 to vps_max_sublayers_minus1, inclusive,            when there is more than one layer in the input bitstream,            and specified be in the range of 0 to            sps_max_sublayers_minus1, inclusive, when there is only one            layer in the input bitstream.    -   2) To solve problem 2, the general sub-bitstream extraction        process is specified such that AUD NAL units are treated in the        same manner as NAL units with nal_unit_type equal to VPS_NUT,        DCI_NUT, or EOB_NUT. In other words, no AUD NAL unit is removed        from the output bitstream outBitstream according to the        nuh_layer_id value.    -   3) To solve problem 3, the general sub-bitstream extraction        process is specified such that it would remove, from the output        bitstream outBitstream, SEI NAL units that contain a scalable        nesting SEI message with sn_ols_flag equal to 0 while the        applicable layers as indicted in the scalable nesting SEI        message do not include any layer in the target OLS.        -   a. In one example, it is specified to remove from            outBitstream all SEI NAL units that contain a scalable            nesting SEI message that has sn_ols_flag equal to 0 and            there is no value in the list nestingLayerId[i] for i in the            range of 0 to nestingNumLayers−1, inclusive, that is in the            list LayerIdInOls targetOlsIdx 1.    -   4) To solve problem 4, the condition “When        LayerIdInOls[targetOlsIdx ] does not include all values of        nuh_layer_id in all NAL units in the bitstream” is changed to be        “When the list LayerIdInOls[targetOlsIdx]does not include all        values of nuh_layer_id in all VCL NAL units in the bitstream        inBitstream”.    -   5) To solve problem 5, the general sub-bitstream extraction        process is specified such that it only extracts scalable-nested        SEI messages from scalable nesting SEI messages with both        sn_ols_flag equal to 1 and sn_subpic_flag equal to 0 to generate        non-scalable-nested SEI messages.    -   6) To solve problem 6, the general sub-bitstream extraction        process is specified such that, when multiple scalable-nested        SEI messages are extracted from one SEI NAL unit seiNalUnitA to        be non-scalable-nested SEI messages, they are still included in        one SEI NAL unit seiNalUnitB in the output bitstream        outBitstream, and the SEI NAL unit seiNalUnitB is included in        the PU that contained the SEI NAL unit seiNalUnitA.    -   7) To solve problem 7, the general sub-bitstream extraction        process is specified such that it removes, from the output        bitstream outBitstream, all SEI NAL units from which some SEI        messages have been extracted and included as non-scalable-nested        SEI messages.        -   a. Alternatively, when the scalable-nested SEI messages in            such an SEI NAL unit apply only to the target OLS (i.e., the            targetOlsIdx-th OLS specified by the VPS), remove the SEI            NAL unit from outBitstream.        -   b. Alternatively, when there is no OLS, other than the            target OLS in the OLSs to which the scalable-nested SEI            messages in such an SEI NAL unit apply, that contains layers            that are all included in the list            LayerIdInOls[targetOlsIdx], remove the SEI NAL unit from            outBitstream.    -   8) To solve problem 8, add constraint such that when an SEI NAL        unit contains an SEI message with payloadType equal to 0, 1, or        130, the SEI NAL unit shall not contain an SEI message with        payloadType not equal to 0 (BP), 1 (PT), 130 (DUI), or 133        (scalable nesting).    -   9) To solve problem 9, the flag        general_same_pic_timing_in_all_ols_flag specifies whether        non-scalable-nested PT and DUI SEI messages apply to all OLSs.        -   a. Alternatively, the flag            general_same_pic_timing_in_all_ols_flag specifies whether            non-scalable-nested BP, PT, and DUI SEI messages apply to            all OLSs.            -   i. In one example, the flag                general_same_pic_timing_in_all_ols_flag is renamed to be                flag general_same_pic_level_hrd_info_in_all_ols_flag,                which specifies whether non-scalable-nested BP, PT, and                DUI SEI messages apply to all OLSs.        -   b. Alternatively, a new flag, e.g., named            general_same_dui_in_all_ols_flag, is added, to specify            whether non-scalable-nested DUI SEI messages apply to all            OLSs.        -   c. Alternatively, a new flag, e.g., named            general_same_bp_in_all_ols_flag, is added, to specify            whether non-scalable-nested BP SEI messages apply to all            OLSs.

6. Embodiments

Below are some example embodiments for some of the invention aspectssummarized above in Section 5, which can be applied to the VVCspecification. The changed texts are based on the latest VVC text inJVET-R2001-vA/v10. Most relevant parts that have been added or modifiedare highlighted in

, and some of the deleted parts are marked with double square brackets(e.g., [[a]] denotes the deletion of the character “a”). There may besome other changes that are editorial in nature and thus nothighlighted.

6.1. First Embodiment

This embodiment is for items 1, 2, 3, 3.a, 4, 5, 6, 7.b, and 8.

C.6 General Sub-Bitstream Extraction Process

Inputs to this process are a bitstream inBitstream, a target OLS indextargetOlsIdx, and a target highest Temporand value tIdTarget.

Output of this process is a sub-bitstream outBitstream.

It is a requirement of bitstream conformance for the input bitstreamthat any output sub-bitstream that satisfies all of the followingconditions shall be a conforming bitstream:

-   -   The output sub-bitstream is the output of the process specified        in this clause with the bitstream, targetOlsIdx equal to an        index to the list of OLSs specified by the VPS, and tIdTarget        equal to any value in the range of 0 to        , inclusive, as inputs.    -   The output sub-bitstream contains at least one VCL NAL unit with        nuh_layer_id equal to each of the nuh_layer_id values in        LayerIdInOls[targetOlsIdx].    -   The output sub-bitstream contains at least one VCL NAL unit with        Temporand equal to tIdTarget.        -   NOTE—A conforming bitstream contains one or more coded slice            NAL units with Temporand equal to 0, but does not have to            contain coded slice NAL units with nuh_layer_id equal to 0.

The output sub-bitstream OutBitstream is derived

:

-   -   1. The bitstream outBitstream is set to be identical to the        bitstream inBitstream.

    -   2. Remove from outBitstream all NAL units with Temporand greater        than tIdTarget.

    -   3. Remove from outBitstream all NAL units with nal_unit_type not        equal to any of DCI_NUT, VPS_NUT,        and EOB_NUT and with nuh_layer_id not included in the list        LayerIdInOls[targetOlsIdx].

    -   4. Remove from outBitstream all VCL NAL units for which all of        the following conditions are true, and        associated non-VCL NAL units that have nal_unit_type equal to        PH_NUT or FD_NUT,        to SUFFIX_SEI_NUT or PREFIX_SEI_NUT        not equal to 0 (BP), 1 (PT), 130 (DUI),        :        -   nal_unit_type is equal to TRAIL_NUT, STSA_NUT, RADL_NUT, or            RASL_NUT, or nal_unit_type is equal to GDR_NUT and the            associated ph_recovery_poc_cnt is not equal to 0.        -   [[nuh_layer_id is equal to LayerIdInOls[targetOlsIdx][j] for            a value of j in the range of 0 to            NumLayersInOls[targetOlsIdx]−1 inclusive.]]        -   TemporalId is greater than or equal to            NumSubLayersInLayerInOLS[targetOlsIdx][GeneralLayerIdx[nuh_layer_id]].

    -   5. Remove from outBitstream all SEI NAL units that contain a        scalable nesting SEI message that has sn_ols_flag equal to 1 and        there is no value of i in the range of 0 to sn_num_olss_minus1,        inclusive, such that NestingOlsIdx[i] is equal to targetOlsIdx.

    -   

    -   7. When LayerIdInOls[targetOlsIdx] does not include all values        of nuh_layer_id in all VCL NAL units in the bitstream        the following applies        -   a. Remove from outBitstream all SEI NAL units that contain a            non-scalable-nested SEI message with payloadType equal to 0            (BP) or 130 (DUI).        -   b. When general_same_pic_timing_in_all_ols_flag is equal to            0, remove from outBitstream all SEI NAL units that contain a            non-scalable-nested SEI message with payloadType equal to 1            (PT).        -   c. When outBitstream contains SEI NAL units that contain a            scalable nesting SEI message with sn_ols_flag equal to 1            that applies to the targetOlsIdx-th OLS (i.e., there is at            least one value of i in the range of 0 to sn_num_olss_minus            1, inclusive, such that NestingOlsIdx[i] is equal to            targetOlsIdx), the following applies            :            -   i. For each scalable-nested BP or DUI SEI message in                such an SEI NAL unit seiNalUnitA,                in outBitstream.            -   ii. When general_same_pic_timing_in_all_ols_flag is                equal to 0, for each scalable-nested PT SEI message in                such an SEI NAL unit seiNalUnitA,                in outBitstream.            -   iii.            -   iv.

D.2.2 General SEI Payload Semantics

It is a requirement of bitstream conformance that the followingrestrictions apply on containing of SEI messages in SEI NAL units:

-   -   When an SEI NAL unit contains a non-scalable-nested BP SEI        message, a non-scalable-nested PT SEI message, or a        non-scalable-nested DUI SEI message, the SEI NAL unit shall not        contain any other SEI message with payloadType not equal to 0        (BP), 1 (PT), or 130 (DUI).

    -   When an SEI NAL unit contains a scalable-nested BP SEI message,        a scalable-nested PT SEI message, or a scalable-nested DUI SEI        message, the SEI NAL unit shall not contain any other SEI        message with payloadType not equal to 0 (BP), 1 (PT), 130 (DUI)        or 133 (scalable nesting).

    -   

FIG. 1 is a block diagram showing an example video processing system1900 in which various embodiments disclosed herein may be implemented.Various embodiments may include some or all of the components of thesystem 1900. The system 1900 may include input 1902 for receiving videocontent. The video content may be received in a raw or uncompressedformat, e.g., 8- or 10-bit multi-component pixel values, or may be in acompressed or encoded format. The input 1902 may represent a networkinterface, a peripheral bus interface, or a storage interface. Examplesof network interface include wired interfaces such as Ethernet, passiveoptical network (PON), etc. and wireless interfaces such as Wi-Fi orcellular interfaces.

The system 1900 may include a coding component 1904 that may implementthe various coding or encoding methods described in the presentdisclosure. The coding component 1904 may reduce the average bitrate ofvideo from the input 1902 to the output of the coding component 1904 toproduce a coded representation of the video. The coding techniques aretherefore sometimes called video compression or video transcodingtechniques. The output of the coding component 1904 may be eitherstored, or transmitted via a communication connected, as represented bythe component 1906. The stored or communicated bitstream (or coded)representation of the video received at the input 1902 may be used bythe component 1908 for generating pixel values or displayable video thatis sent to a display interface 1910. The process of generatinguser-viewable video from the bitstream representation is sometimescalled video decompression. Furthermore, while certain video processingoperations are referred to as “coding” operations or tools, it will beappreciated that the coding tools or operations are used at an encoderand corresponding decoding tools or operations that reverse the resultsof the coding will be performed by a decoder.

Examples of a peripheral bus interface or a display interface mayinclude universal serial bus (USB) or high definition multimediainterface (HDMI) or DisplayPort, and so on. Examples of storageinterfaces include serial advanced technology attachment (SATA),peripheral component interface (PCI), integrated drive electronics (IDE)interface, and the like. The embodiments described in the presentdisclosure may be embodied in various electronic devices such as mobilephones, laptops, smartphones or other devices that are capable ofperforming digital data processing and/or video display.

FIG. 2 is a block diagram of a video processing apparatus 3600. Theapparatus 3600 may be used to implement one or more of the methodsdescribed herein. The apparatus 3600 may be embodied in a smartphone,tablet, computer, Internet of Things (IoT) receiver, and so on. Theapparatus 3600 may include one or more processors 3602, one or morememories 3604 and video processing hardware 3606. The processor(s) 3602may be configured to implement one or more methods described in thepresent disclosure. The memory (memories) 3604 may be used for storingdata and code used for implementing the methods and embodimentsdescribed herein. The video processing hardware 3606 may be used toimplement, in hardware circuitry, some embodiments described in thepresent disclosure.

FIG. 4 is a block diagram that illustrates an example video codingsystem 100 that may utilize the embodiments of this disclosure.

As shown in FIG. 4 , video coding system 100 may include a source device110 and a destination device 120. Source device 110 generates encodedvideo data which may be referred to as a video encoding device.Destination device 120 may decode the encoded video data generated bysource device 110 which may be referred to as a video decoding device.

Source device 110 may include a video source 112, a video encoder 114,and an input/output (I/O) interface 116.

Video source 112 may include a source such as a video capture device, aninterface to receive video data from a video content provider, and/or acomputer graphics system for generating video data, or a combination ofsuch sources. The video data may comprise one or more pictures. Videoencoder 114 encodes the video data from video source 112 to generate abitstream. The bitstream may include a sequence of bits that form acoded representation of the video data. The bitstream may include codedpictures and associated data. The coded picture is a codedrepresentation of a picture. The associated data may include sequenceparameter sets, picture parameter sets, and other syntax structures. I/Ointerface 116 may include a modulator/demodulator (modem) and/or atransmitter. The encoded video data may be transmitted directly todestination device 120 via I/O interface 116 through network 130 a. Theencoded video data may also be stored onto a storage medium/server 130 bfor access by destination device 120.

Destination device 120 may include an I/O interface 126, a video decoder124, and a display device 122.

I/O interface 126 may include a receiver and/or a modem. I/O interface126 may acquire encoded video data from the source device 110 or thestorage medium/ server 130 b. Video decoder 124 may decode the encodedvideo data. Display device 122 may display the decoded video data to auser. Display device 122 may be integrated with the destination device120, or may be external to destination device 120 which be configured tointerface with an external display device.

Video encoder 114 and video decoder 124 may operate according to a videocompression standard, such as the High Efficiency Video Coding (HEVC)standard, Versatile Video Coding (VVC) standard and other current and/orfurther standards.

FIG. 5 is a block diagram illustrating an example of video encoder 200,which may be video encoder 114 in the system 100 illustrated in FIG. 4 .

Video encoder 200 may be configured to perform any or all of theembodiments of this disclosure. In the example of FIG. 5 , video encoder200 includes a plurality of functional components. The embodimentsdescribed in this disclosure may be shared among the various componentsof video encoder 200. In some examples, a processor may be configured toperform any or all of the embodiments described in this disclosure.

The functional components of video encoder 200 may include a partitionunit 201; a prediction unit 202, which may include a mode select unit203, a motion estimation unit 204, a motion compensation unit 205, andan intra prediction unit 206; a residual generation unit 207; atransform unit 208; a quantization unit 209; an inverse quantizationunit 210; an inverse transform unit 211; a reconstruction unit 212; abuffer 213; and an entropy encoding unit 214.

In other examples, video encoder 200 may include more, fewer, ordifferent functional components. In an example, prediction unit 202 mayinclude an intra block copy (IBC) unit. The IBC unit may performprediction in an IBC mode in which at least one reference picture is apicture where the current video block is located.

Furthermore, some components, such as motion estimation unit 204 andmotion compensation unit 205 may be highly integrated, but arerepresented in the example of FIG. 5 separately for purposes ofexplanation.

Partition unit 201 may partition a picture into one or more videoblocks. Video encoder 200 and video decoder 300 may support variousvideo block sizes.

Mode select unit 203 may select one of the coding modes, intra or inter,e.g., based on error results, and provide the resulting intra- orinter-coded block to a residual generation unit 207 to generate residualblock data and to a reconstruction unit 212 to reconstruct the encodedblock for use as a reference picture. In some examples, mode select unit203 may select a combination of intra and inter prediction (CIIP) modein which the prediction is based on an inter prediction signal and anintra prediction signal. Mode select unit 203 may also select aresolution for a motion vector (e.g., a sub-pixel or integer pixelprecision) for the block in the case of inter-prediction.

To perform inter prediction on a current video block, motion estimationunit 204 may generate motion information for the current video block bycomparing one or more reference frames from buffer 213 to the currentvideo block. Motion compensation unit 205 may determine a predictedvideo block for the current video block based on the motion informationand decoded samples of pictures from buffer 213 other than the pictureassociated with the current video block.

Motion estimation unit 204 and motion compensation unit 205 may performdifferent operations for a current video block, for example, dependingon whether the current video block is in an I slice, a P slice, or a Bslice.

In some examples, motion estimation unit 204 may perform uni-directionalprediction for the current video block, and motion estimation unit 204may search reference pictures of list 0 or list 1 for a reference videoblock for the current video block. Motion estimation unit 204 may thengenerate a reference index that indicates the reference picture in list0 or list 1 that contains the reference video block and a motion vectorthat indicates a spatial displacement between the current video blockand the reference video block. Motion estimation unit 204 may output thereference index, a prediction direction indicator, and the motion vectoras the motion information of the current video block. Motioncompensation unit 205 may generate the predicted video block of thecurrent block based on the reference video block indicated by the motioninformation of the current video block.

In other examples, motion estimation unit 204 may perform bi-directionalprediction for the current video block, motion estimation unit 204 maysearch the reference pictures in list 0 for a reference video block forthe current video block and may also search the reference pictures inlist 1 for another reference video block for the current video block.Motion estimation unit 204 may then generate reference indexes thatindicate the reference pictures in list 0 and list 1 containing thereference video blocks and motion vectors that indicate spatialdisplacements between the reference video blocks and the current videoblock. Motion estimation unit 204 may output the reference indexes andthe motion vectors of the current video block as the motion informationof the current video block. Motion compensation unit 205 may generatethe predicted video block of the current video block based on thereference video blocks indicated by the motion information of thecurrent video block.

In some examples, motion estimation unit 204 may output a full set ofmotion information for decoding processing of a decoder.

In some examples, motion estimation unit 204 may not output a full setof motion information for the current video. Rather, motion estimationunit 204 may signal the motion information of the current video blockwith reference to the motion information of another video block. Forexample, motion estimation unit 204 may determine that the motioninformation of the current video block is sufficiently similar to themotion information of a neighboring video block.

In one example, motion estimation unit 204 may indicate, in a syntaxstructure associated with the current video block, a value thatindicates to the video decoder 300 that the current video block has thesame motion information as the another video block.

In another example, motion estimation unit 204 may identify, in a syntaxstructure associated with the current video block, another video blockand a motion vector difference (MVD). The motion vector differenceindicates a difference between the motion vector of the current videoblock and the motion vector of the indicated video block. The videodecoder 300 may use the motion vector of the indicated video block andthe motion vector difference to determine the motion vector of thecurrent video block.

As discussed above, video encoder 200 may predictively signal the motionvector. Two examples of predictive signalling techniques that may beimplemented by video encoder 200 include advanced motion vectorprediction (AMVP) and merge mode signalling.

Intra prediction unit 206 may perform intra prediction on the currentvideo block. When intra prediction unit 206 performs intra prediction onthe current video block, intra prediction unit 206 may generateprediction data for the current video block based on decoded samples ofother video blocks in the same picture. The prediction data for thecurrent video block may include a predicted video block and varioussyntax elements.

Residual generation unit 207 may generate residual data for the currentvideo block by subtracting (e.g., indicated by the minus sign) thepredicted video block(s) of the current video block from the currentvideo block. The residual data of the current video block may includeresidual video blocks that correspond to different sample components ofthe samples in the current video block.

In other examples, there may be no residual data for the current videoblock for the current video block, for example in a skip mode, andresidual generation unit 207 may not perform the subtracting operation.

Transform processing unit 208 may generate one or more transformcoefficient video blocks for the current video block by applying one ormore transforms to a residual video block associated with the currentvideo block.

After transform processing unit 208 generates a transform coefficientvideo block associated with the current video block, quantization unit209 may quantize the transform coefficient video block associated withthe current video block based on one or more quantization parameter (QP)values associated with the current video block.

Inverse quantization unit 210 and inverse transform unit 211 may applyinverse quantization and inverse transforms to the transform coefficientvideo block, respectively, to reconstruct a residual video block fromthe transform coefficient video block. Reconstruction unit 212 may addthe reconstructed residual video block to corresponding samples from oneor more predicted video blocks generated by the prediction unit 202 toproduce a reconstructed video block associated with the current blockfor storage in the buffer 213.

After reconstruction unit 212 reconstructs the video block, loopfiltering operation may be performed reduce video blocking artifacts inthe video block.

Entropy encoding unit 214 may receive data from other functionalcomponents of the video encoder 200. When entropy encoding unit 214receives the data, entropy encoding unit 214 may perform one or moreentropy encoding operations to generate entropy encoded data and outputa bitstream that includes the entropy encoded data.

FIG. 6 is a block diagram illustrating an example of video decoder 300which may be video decoder 124 in the system 100 illustrated in FIG. 4 .

The video decoder 300 may be configured to perform any or all of theembodiments of this disclosure. In the example of FIG. 6 , the videodecoder 300 includes a plurality of functional components. Theembodiments described in this disclosure may be shared among the variouscomponents of the video decoder 300. In some examples, a processor maybe configured to perform any or all of the embodiments described in thisdisclosure.

In the example of FIG. 6 , video decoder 300 includes an entropydecoding unit 301, a motion compensation unit 302, an intra predictionunit 303, an inverse quantization unit 304, an inverse transformationunit 305, a reconstruction unit 306, and a buffer 307. Video decoder 300may, in some examples, perform a decoding pass generally reciprocal tothe encoding pass described with respect to video encoder 200 (FIG. 5 ).

Entropy decoding unit 301 may retrieve an encoded bitstream. The encodedbitstream may include entropy coded video data (e.g., encoded blocks ofvideo data). Entropy decoding unit 301 may decode the entropy codedvideo data, and from the entropy decoded video data, motion compensationunit 302 may determine motion information including motion vectors,motion vector precision, reference picture list indexes, and othermotion information. Motion compensation unit 302 may, for example,determine such information by performing the AMVP and merge mode.

Motion compensation unit 302 may produce motion compensated blocks,possibly performing interpolation based on interpolation filters.Identifiers for interpolation filters to be used with sub-pixelprecision may be included in the syntax elements.

Motion compensation unit 302 may use interpolation filters as used byvideo encoder 200 during encoding of the video block to calculateinterpolated values for sub-integer pixels of a reference block. Motioncompensation unit 302 may determine the interpolation filters used byvideo encoder 200 according to received syntax information and use theinterpolation filters to produce predictive blocks.

Motion compensation unit 302 may use some of the syntax information todetermine sizes of blocks used to encode frame(s) and/or slice(s) of theencoded video sequence, partition information that describes how eachmacroblock of a picture of the encoded video sequence is partitioned,modes indicating how each partition is encoded, one or more referenceframes (and reference frame lists) for each inter-encoded block, andother information to decode the encoded video sequence.

Intra prediction unit 303 may use intra prediction modes for examplereceived in the bitstream to form a prediction block from spatiallyadjacent blocks. Inverse quantization unit 304 inverse quantizes, i.e.,de-quantizes, the quantized video block coefficients provided in thebitstream and decoded by entropy decoding unit 301. Inversetransformation unit 305 applies an inverse transform.

Reconstruction unit 306 may sum the residual blocks with thecorresponding prediction blocks generated by motion compensation unit302 or intra prediction unit 303 to form decoded blocks. If desired, adeblocking filter may also be applied to filter the decoded blocks inorder to remove blockiness artifacts. The decoded video blocks are thenstored in buffer 307, which provides reference blocks for subsequentmotion compensation/intra prediction and also produces decoded video forpresentation on a display device.

A listing of solutions describes some embodiments of the presentdisclosure.

A first set of solutions is provided next. The following solutions showexample embodiments discussed in the previous section (e.g., items 1-9).

1. A method of video processing (e.g., method 600 in FIG. 3 ),comprising performing (602) a conversion between a video comprising oneor more video layers comprising one or more video pictures and a codedrepresentation of the video, wherein the coded representation conformsto a format rule related to extraction of a sub-bitstream from the codedrepresentation.

2. The method of solution 1, further comprising: extracting thesub-bitstream from the coded representation according to the formatrule.

The following solutions show example embodiments discussed in theprevious section (e.g., item 1)

3. The method of any of solutions 1-2, wherein, during extracting thesub-bitstream, a target identifier used for the extracting is allowed tobe between range 0 to a value of a syntax field indicating in a videoparameter set for the coded representation.

The following solutions show example embodiments discussed in theprevious section (e.g., item 2)

4. The method of any of solutions 1-3, wherein the sub-bitstream isextracted without removing an access unit delimiter network abstractionlayer (AUD NAL) from an output bitstream according to a layeridentifier.

The following solutions show example embodiments discussed in theprevious section (e.g., item 3)

5. The method of any of solutions 1-4, wherein the sub-bitstream isextracted by selectively removing network abstraction layer units thatinclude a scalable nesting supplementary enhancement information messagethat are not applicable to output layers being extracted.

The following solutions show example embodiments discussed in theprevious section (e.g., item 5)

6. The method of any of solutions 1-5, wherein the sub-bitstream isextracted by constraining the extracting to generate a non-scalablenested supplemental enhancement information (SEI) from a scalablenesting SEI message using a flag for output layer set being set and aflag for subpicture being disabled.

The following solutions show example embodiments discussed in theprevious section (e.g., item 6)

7. The method of any of solutions 1-6, wherein the sub-bitstream isextracted according to a rule specifying extraction of multiplescalable-nested supplemental enhancement information (SEI) messages froma single SEI network abstraction layer unit.

The following solutions show example embodiments discussed in theprevious section (e.g., item 7)

8. The method of any of solutions 1-7, wherein the sub-bitstream isextracted according to a rule that removes, from the codedrepresentation, all supplemental enhancement information (SEI) networkabstraction layer (NAL) units from which some SEI messages have beenextracted and included as non-scalable-nested SEI messages.

The following solutions show example embodiments discussed in theprevious section (e.g., item 8)

9. The method of any of solutions 1-8, wherein the format rule specifiesthat when a supplemental enhancement information network abstractionlayer (SEI NAL) unit contains an SEI message with payloadType equal to0, 1, or 130, the SEI NAL unit is not allowed to contain an SEI messagewith payloadType not equal to 0 (BP), 1 (PT), 130 (DUI), or 133(scalable nesting).

10. The method of any of solutions 1-9, wherein the performing theconversion comprises encoding the video into the coded representation.

11. The method of any of solutions 1-9, wherein the performing theconversion comprises parsing and decoding the coded representation togenerate the video.

12. A video decoding apparatus comprising a processor configured toimplement a method recited in one or more of solutions 1 to 11.

13. A video encoding apparatus comprising a processor configured toimplement a method recited in one or more of solutions 1 to 11.

14. A computer program product having computer code stored thereon, thecode, when executed by a processor, causes the processor to implement amethod recited in any of solutions 1 to 11.

15. A method, apparatus or system described in the present disclosure.

A second set of solutions show example embodiments discussed in theprevious section (e.g., items 1, 4, 8, and 9).

1. A method of video processing (e.g., method 700 as shown in FIG. 7A),comprising: performing 702 a conversion between a video comprisingmultiple layers and a bitstream of the video according to a rule,wherein the rule specifies a maximum allowed value of a temporal layeridentifier value of a sub-bitstream that is extracted from thebitstream.

2. The method of solution 1, wherein the rule is responsive to a numberof layers in the bitstream.

3. The method of solution 1 or 2, wherein, in case that the number oflayers is greater than 1, the rule specifies that the maximum allowedvalue of the temporal layer identifier is in a range of zero to a valuebased on a first syntax element in a video parameter set referenced bythe bitstream.

4. The method of solution 1 or 2, wherein, in case that the number oflayers is equal to 1, the rule specifies that the maximum allowed valueof the temporal layer identifier is in a range of zero to a value basedon a second syntax element in a sequence parameter set referenced by thebitstream.

5. The method of solution 3, wherein the first syntax element specifiesa maximum number of temporal sublayers that are allowed to be present ina layer specified by the video parameter set minus 1.

6. The method of solution 4, wherein the second syntax element specifiesa maximum number of temporal sublayers that are allowed to be present ina layer specified by the sequence parameter set minus 1.

7. The method of solution 3, wherein the first syntax element in thevideo parameter set is vps_max_sublayers_minus1.

8. The method of solution 4, wherein the second syntax element in thesequence parameter set is sps_max_sublayers_minus1.

9. The method of solution 3 or 4, wherein the value is equal to a valueof the first syntax element or the second syntax element.

10. A method of video processing (e.g., method 710 as shown in FIG. 7B),comprising: performing 712 a conversion between a video and a bitstreamof the video according to a rule, wherein the rule defines networkabstraction layer (NAL) units to be extracted from the bitstream duringa sub-bitstream extraction process to output a sub-bitstream, andwherein the rule specifies to derive the sub-bitstream based on whethera list of NAL unit header identifier values in an output layer set (OLS)with a target OLS index does not include all values of NAL unit headeridentifiers in all video coding layer (VCL) NAL units in the bitstreamthat is input to the sub -bitstream extraction process.

11. The method of solution 10, wherein the rule specifies to remove fromthe sub-bitstream all supplemental enhancement information (SEI) NALunits that contain a non-scalable-nested SEI message with payload typeequal to 0 or 130, in case that the list of NAL unit header identifiervalues in the output layer set does not include all values of NAL unitheader identifiers in all VCL NAL units in the bitstream.

12. A method of video processing (e.g., method 720 as shown in FIG. 7C),comprising: performing 722 a conversion between a video and a bitstreamof the video according to a rule, wherein the rule defines networkabstraction layer (NAL) units to be extracted from the bitstream duringa sub-bitstream extraction process to output a sub-bitstream, andwherein the rule specifies, responsive to a payload type of a firstsupplemental enhancement information (SEI) message included in an SEInetwork abstraction layer (NAL) unit, to disallow the SEI NAL unit tocontain an SEI message with a certain payload type.

13. The method of solution 12, wherein the payload type of the first SEImessage is 0, 1, or 130 and the rule specifies to disallow the SEI NALunit to contain the SEI message with the certain payload type that isnot equal to 0, 1, 130, or 133.

14. A method of video processing (e.g., method 730 as shown in FIG. 7D),comprising: performing 732 a conversion between a video and a bitstreamof the video according to a format rule, wherein the format rulespecifies that the bitstream includes a flag specifying whether one ormore non-scalable nested supplemental enhancement information (SEI)messages with one or more particular payload types apply to all outputlayer sets referenced by a coding layer.

15. The method of solution 14, wherein the one or more particularpayload types are equal to 1 and 130.

16. The method of solution 14, wherein the one or more non-scalable SEImessages are picture timing (PT) SEI messages and decoding unitinformation (DUI) SEI messages.

17. The method of solution 14, wherein the one or more particularpayload types are equal to 0, 1 and 130.

18. The method of solution 14, wherein the one or more non-scalable SEImessages are buffering period (BP) SEI messages and decoding unitinformation (DUI) SEI messages.

19. The method of solution 14, wherein the one or more particularpayload type is equal to 130.

20. The method of solution 14, wherein the one or more non-scalable SEImessages are decoding unit information (DUI) SEI messages.

21. The method of solution 14, wherein the one or more particularpayload type is equal to 0.

22. The method of solution 1, wherein the one or more non-scalable SEImessages are buffering period (BP) SEI messages.

23. The method of any of solutions 1 to 22, wherein the conversionincludes encoding the video into the bitstream.

24. The method of any of solutions 1 to 22, wherein the conversionincludes decoding the video from the bitstream.

25. The method of any of solutions 1 to 22, wherein the conversionincludes generating the bitstream from the video, and the method furthercomprises: storing the bitstream in a non-transitory computer-readablerecording medium.

26. A video processing apparatus comprising a processor configured toimplement a method recited in any one or more of solutions 1 to 25.

27. A method of storing a bitstream of a video, comprising, a methodrecited in any one of solutions 1 to 25, and further including storingthe bitstream to a non-transitory computer-readable recording medium.

28. A computer readable medium storing program code that, when executed,causes a processor to implement a method recited in any one or more ofsolutions 1 to 25.

29. A computer readable medium that stores a bitstream generatedaccording to any of the above described methods.

30. A video processing apparatus for storing a bitstream representation,wherein the video processing apparatus is configured to implement amethod recited in any one or more of solutions 1 to 25.

A third set of solutions show example embodiments discussed in theprevious section (e.g., item 2).

1. A method of video processing (e.g., method 800 as shown in FIG. 8 ),comprising: performing 802 a conversion between a video and a bitstreamof the video according to a rule, wherein the rule specifies asub-bitstream extraction process by which an output sub-bitstream isextracted from the bitstream without removing network abstraction layer(NAL) units of a particular type and having a particular NAL unit headeridentifier value, wherein the particular type includes an access unitdelimiter (AUD) NAL unit.

2. The method of solution 1, wherein the particular type includes avideo parameter set (VPS) NAL unit.

3. The method of solution 1 or 2, wherein the particular type includes adecoding capability information NAL unit.

4. The method of any of solutions 1-3, wherein the particular typeincludes an end of bitstream NAL unit.

5. The method of any of solutions 1-4, wherein the particular typeincludes a supplemental enhancement information NAL unit containing anon-scalable nested SEI massage with a payload type that is equal to 0,1, 130 or 203.

6. The method of any of solutions 1-6, wherein the particular NAL unitheader identifier value includes a layer identifier value that isincluded in a list of layer values for the output sub-bitstream.

7. The method of solution 1, wherein the rule specifies that aparticular type NAL unit is not removed from the output bitstreamaccording to the exception regardless of a NAL unit header identifiervalue associated with the particular type NAL unit.

8. The method of any of solutions 1 to 7, wherein the conversionincludes encoding the video into the bitstream.

9. The method of any of solutions 1 to 7, wherein the conversionincludes decoding the video from the bitstream.

10. The method of any of solutions 1 to 7, wherein the conversionincludes generating the bitstream from the video, and the method furthercomprises: storing the bitstream in a non-transitory computer-readablerecording medium.

11. A video processing apparatus comprising a processor configured toimplement a method recited in any one or more of solutions 1 to 10.

12. A method of storing a bitstream of a video, comprising, a methodrecited in any one of solutions 1 to 10, and further including storingthe bitstream to a non-transitory computer-readable recording medium.

13. A computer readable medium storing program code that, when executed,causes a processor to implement a method recited in any one or more ofsolutions 1 to 10.

14. A computer readable medium that stores a bitstream generatedaccording to any of the above described methods.

15. A video processing apparatus for storing a bitstream representation,wherein the video processing apparatus is configured to implement amethod recited in any one or more of solutions 1 to 10.

A fourth set of solutions show example embodiments discussed in theprevious section (e.g., item 3).

1. A method of video processing (e.g., method 900 as shown in FIG. 9 ),comprising: performing 902 a conversion between a video including one ormore layers and a bitstream of the video according to a rule, whereinthe rule specifies, in a process of sub-bitstream extraction, to removenetwork abstraction layer (NAL) units that include a scalable nestingsupplementary enhancement information (SEI) message applied to layersthat are not included in a target output layer set (OLS).

2. The method of solution 1, wherein the scalable nesting SEI message isassociated with a flag having a value equal to a particular value thatspecifies the scalable nesting SEI message applies to the layers.

3. The method of solution 2, wherein the particular value is 0.

4. The method of solution 1, wherein the rule further specifies toremove the NAL units that include the SEI message having no value in afirst list (NestingLayerID[i]) that is in a second list(LayerIdInOls[targetOlsIdx]), whereby the first list specifies NAL unitheader identifier values of layers to which the scalable nesting SEImessage applies and i is in a range of 0 to NumLayers−1, NumLayersindicating a number of the layers to which the scalable nesting SEImessage applies, and the second list specifies the NAL unit headeridentifier values in the target output layer set with a target outputlayer index.

5. The method of any of solutions 1 to 4, wherein the conversionincludes encoding the video into the bitstream.

6. The method of any of solutions 1 to 4, wherein the conversionincludes decoding the video from the bitstream.

7. The method of any of solutions 1 to 4, wherein the conversionincludes generating the bitstream from the video, and the method furthercomprises: storing the bitstream in a non-transitory computer-readablerecording medium.

8. A video processing apparatus comprising a processor configured toimplement a method recited in any one or more of solutions 1 to 7.

9. A method of storing a bitstream of a video, comprising, a methodrecited in any one of solutions 1 to 7, and further including storingthe bitstream to a non-transitory computer-readable recording medium.

10. A computer readable medium storing program code that, when executed,causes a processor to implement a method recited in any one or more ofsolutions 1 to 7.

11. A computer readable medium that stores a bitstream generatedaccording to any of the above described methods.

12. A video processing apparatus for storing a bitstream representation,wherein the video processing apparatus is configured to implement amethod recited in any one or more of solutions 1 to 7.

A fifth set of solutions show example embodiments discussed in theprevious section (e.g., items 5-7).

1. A method of video processing (e.g., method 1000 as shown in FIG.10A), comprising: performing 1002 a conversion between a video includingone or more layers and a bitstream of the video according to a rule,wherein the rule specifies, in a process of sub-bitstream extraction, anon-scalable nested supplementary enhancement information (SEI) messageis generated by extracting an SEI message that is scalably-nested from ascalable nesting SEI message based on a first flag indicating whetherthe SEI message applies to specific output layer sets (OLSs) and asecond flag indicating whether the SEI message applies to allsubpictures or only to specific subpictures.

2. The method of solution 1, wherein the rule specifies to generate thenon-scalable nested SEI message responsive to conditions i) the firstflag having a first value that specifies that the SEI message applies tospecific output layer sets (OLSs) and ii) the second flag having asecond value that specifies that the SEI message that applies to thespecific OLSs applies to all subpictures of the specified OLS s beingsatisfied.

3. A method of video processing (e.g., method 1010 as shown in FIG.10B), comprising: performing 1012 a conversion between a video includingone or more layers and a bitstream of the video according to a rule,wherein the rule specifies, in a process of sub-bitstream extraction, anon-scalable nested supplemental enhancement information (SEI) messageis generated by extracting multiple scalable-nested SEI messages from afirst SEI network abstraction layer (NAL) unit in a picture unit.

4. The method of solution 3, wherein the multiple scalable-nested SEImessages are included in a second SEI NAL unit included in the pictureunit.

5. The method of solution 4, wherein the second SEI NAL unit isimmediately after the first SEI NAL unit.

6. A method of video processing (e.g., method 1020 as shown in FIG.10C), comprising: performing 1022 a conversion between a video includingone or more layers and a bitstream of the video according to a rule,wherein the rule specifies a sub-bitstream extraction process togenerate an output bitstream, wherein the rule specifies handling of oneor more supplemental enhancement information (SEI) network abstractionlayer (NAL) units during the sub-bitstream extraction process.

7. The method of solution 6, wherein the rule specifies to remove theone or more SEI NAL units from which some SEI messages have beenextracted and included as non-scalable nested SEI messages.

8. The method of solution 6, wherein the rule specifies to remove a SEINAL unit that contains a scalable-nested SEI message applying only to atarget output layer set.

9. The method of solution 6, wherein the rule specifies to remove a SEINAL unit that contains a scalable nested SEI message that applies tooutput layer sets (OLSs) in which there is no OLS other than a targetOLS.

10. The method of any of solutions 1 to 9, wherein the conversionincludes encoding the video into the bitstream.

11. The method of any of solutions 1 to 9, wherein the conversionincludes decoding the video from the bitstream.

12. The method of any of solutions 1 to 9, wherein the conversionincludes generating the bitstream from the video, and the method furthercomprises: storing the bitstream in a non-transitory computer-readablerecording medium.

13. A video processing apparatus comprising a processor configured toimplement a method recited in any one or more of solutions 1 to 12.

14. A method of storing a bitstream of a video, comprising, a methodrecited in any one of solutions 1 to 12, and further including storingthe bitstream to a non-transitory computer-readable recording medium.

15. A computer readable medium storing program code that, when executed,causes a processor to implement a method recited in any one or more ofsolutions 1 to 12.

16. A computer readable medium that stores a bitstream generatedaccording to any of the above described methods.

17. A video processing apparatus for storing a bitstream representation,wherein the video processing apparatus is configured to implement amethod recited in any one or more of solutions 1 to 12.

The disclosed and other solutions, examples, embodiments, modules andthe functional operations described in this disclosure can beimplemented in digital electronic circuitry, or in computer software,firmware, or hardware, including the structures disclosed in thisdisclosure and their structural equivalents, or in combinations of oneor more of them. The disclosed and other embodiments can be implementedas one or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a computer readable medium forexecution by, or to control the operation of, data processing apparatus.The computer readable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more them. The term “data processing apparatus” encompassesall apparatus, devices, and machines for processing data, including byway of example a programmable processor, a computer, or multipleprocessors or computers. The apparatus can include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, or acombination of one or more of them. A propagated signal is anartificially generated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal, that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this disclosure can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., a field-programmable gate array (FPGA) or anapplication-specific integrated circuit (ASIC).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random-access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Computer readable media suitable for storingcomputer program instructions and data include all forms of non-volatilememory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., erasable programmable read-onlymemory (EPROM), electronically erasable programmable read-only memory(EEPROM), and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto optical disks; and compact disc,read-only memory (CD-ROM) and digital versatile disc, read-only memory(DVD-ROM) disks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

While the present disclosure contains many specifics, these should notbe construed as limitations on the scope of any subject matter or ofwhat may be claimed, but rather as descriptions of features that may bespecific to particular embodiments of the present disclosure. Certainfeatures that are described in the present disclosure in the context ofseparate embodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Moreover, the separation of various system components in theembodiments described in the present disclosure should not be understoodas requiring such separation in all embodiments.

Only a few embodiments and examples are described and other embodiments,enhancements and variations can be made based on what is described andillustrated in the present disclosure.

What is claimed is:
 1. A method of processing video data, comprising:performing a conversion between a video including one or more layers anda bitstream of the video according to a rule, wherein the rule specifiesthat, in a first process of sub-bitstream extraction to output a firstoutput sub-bitstream, the first output sub-bitstream is extractedwithout removing network abstraction layer (NAL) units of a particulartype and having a particular NAL unit header identifier value, andwherein the particular type includes an access unit delimiter (AUD) NALunit.
 2. The method of claim 1, wherein the particular type includes atleast one of the following: a video parameter set (VPS) NAL unit, adecoding capability information NAL unit, an end of bitstream NAL unit.3. The method of claim 1, wherein the particular type includes asupplemental enhancement information (SEI) NAL unit containing anon-scalable nested SEI massage with a payload type that is equal to 0,1, 130 or
 203. 4. The method of claim 1, wherein the particular NAL unitheader identifier value includes a layer identifier value that isincluded in a list of layer values for the first output sub-bitstream.5. The method of claim 1, wherein the rule further specifies that, a NALunit of the particular type is not removed from the first outputsub-bitstream regardless of a NAL unit header identifier valueassociated with the NAL unit of the particular type.
 6. The method ofclaim 1, wherein the rule further specifies that, when an SEI NAL unitcontains a scalable-nested SEI message with a payload type equal to 0(buffering period (BP)), 1 (picture timing (PT)), 130 (decoding unitinformation (DUI)), or 203 (subpicture level information (SLI)), the SEINAL unit excludes any other SEI message with a payload type not equal to0 (BP), 1 (PT), 130 (DUI), 203 (SLI), and 133 (scalable nesting).
 7. Themethod of claim 1, wherein the rule further specifies that, in a secondprocess of sub-bitstream extraction to output a second outputsub-bitstream, a maximum allowed value of a temporal layer identifier ofthe second output sub-bitstream is in a range of zero to a value basedon a first syntax element in a video parameter set referenced by thebitstream, and wherein the first syntax element specifies a maximumnumber of temporal sublayers that are allowed to be present in a layerspecified by the video parameter set minus
 1. 8. The method of claim 1,wherein the rule further specifies, in the first process ofsub-bitstream extraction, to remove NAL units that include a scalablenesting SEI message applied to layers that are not included in a targetoutput layer set (OLS), wherein the scalable nesting SEI message isassociated with a flag having a value equal to a particular value thatspecifies the scalable nesting SEI message applies to the layers, andwherein the rule further specifies to remove the NAL units that includethe SEI message having no value in a first list (NestingLayerIDN) thatis in a second list (LayerIdInOls[targetOlsIdx]), wherein the first listspecifies NAL unit header identifier values of layers to which thescalable nesting SEI message applies and i is in a range of 0 toNumLayers−1, NumLayers indicates a number of the layers to which thescalable nesting SEI message applies, and the second list specifies theNAL unit header identifier values in the target output layer set with atarget output layer index.
 9. The method of claim 8, wherein theparticular value is
 0. 10. The method of claim 1, wherein the conversionincludes encoding the video into the bitstream.
 11. The method of claim1, wherein the conversion includes decoding the video from thebitstream.
 12. An apparatus for processing video data comprising aprocessor and a non-transitory memory with instructions thereon, whereinthe instructions upon execution by the processor, cause the processorto: perform a conversion between a video including one or more layersand a bitstream of the video according to a rule, wherein the rulespecifies that, in a first process of sub-bitstream extraction to outputa first output sub-bitstream, the first output sub-bitstream isextracted without removing network abstraction layer (NAL) units of aparticular type and having a particular NAL unit header identifiervalue, and wherein the particular type includes an access unit delimiter(AUD) NAL unit.
 13. The apparatus of claim 12, wherein the particulartype includes at least one of the following: a video parameter set (VPS)NAL unit, a decoding capability information NAL unit, an end ofbitstream NAL unit.
 14. The apparatus of claim 12, wherein theparticular type includes a supplemental enhancement information (SEI)NAL unit containing a non-scalable nested SEI massage with a payloadtype that is equal to 0, 1, 130, or
 203. 15. The apparatus of claim 12,wherein the particular NAL unit header identifier value includes a layeridentifier value that is included in a list of layer values for thefirst output sub-bitstream.
 16. A non-transitory computer-readablestorage medium storing instructions that cause a processor to: perform aconversion between a video including one or more layers and a bitstreamof the video according to a rule, wherein the rule specifies that, in afirst process of sub-bitstream extraction to output a first outputsub-bitstream, the first output sub-bitstream is extracted withoutremoving network abstraction layer (NAL) units of a particular type andhaving a particular NAL unit header identifier value, and wherein theparticular type includes an access unit delimiter (AUD) NAL unit. 17.The non-transitory computer-readable storage medium of claim 16, whereinthe particular type includes at least one of the following: a videoparameter set (VPS) NAL unit, a decoding capability information NALunit, an end of bitstream NAL unit.
 18. A non-transitorycomputer-readable recording medium storing a bitstream of a video whichis generated by a method performed by a video processing apparatus,wherein the method comprises: generating the bitstream of the videoincluding one or more layers according to a rule, wherein the rulespecifies that, in a first process of sub-bitstream extraction to outputa first output sub-bitstream, the first output sub-bitstream isextracted without removing network abstraction layer (NAL) units of aparticular type and having a particular NAL unit header identifiervalue, and wherein the particular type includes an access unit delimiter(AUD) NAL unit.
 19. The non-transitory computer-readable recordingmedium of claim 18, wherein the particular type includes at least one ofthe following: a video parameter set (VPS) NAL unit, a decodingcapability information NAL unit, an end of bitstream NAL unit.
 20. Thenon-transitory computer-readable recording medium of claim 18, whereinthe particular type includes a supplemental enhancement information(SEI) NAL unit containing a non-scalable nested SEI massage with apayload type that is equal to 0, 1, 130, or 203.